1. Technical Field of the Invention
This invention relates generally to deblurring of still images such as those captured by digital cameras, and more specifically to deblurring still images generated as a multi-frame full resolution image and especially those produced by a camera having a rolling shutter, meaning that the pixels of the frame are captured at different times.
2. Background Art
A digital camera includes a lens mechanism having a fixed or variable shutter and a fixed or variable aperture, and a sensor having a fixed or variable sensitivity. The terms ISO and SOS refer to different but related methods of measuring the effective, useable speed of a sensor. This disclosure will refer simply to these as “sensor speed” so as to not specifically distinguish between ISO and SOS characteristics. The term “shutter speed” will refer to the duration of the opening of the shutter.
Adjusting a digital camera's sensor speed upward enables the use of a faster shutter speed but tends to increase the noisiness of the resulting image, and adjusting it downward tends to decrease the noisiness but increases the required exposure duration.
Slow shutter speeds tend to result in blurred images, caused by motion of the camera and/or the photographed subject. This motion causes a time-dependent smearing in which the photons captured from a specific subject location fall upon different sensor locations. This will be referred to as “motion blur”.
Some cameras have been adapted to reduce noise and motion blur by using what may be termed “multi-frame compositing”. Rather than capturing a photograph by taking a single frame at e.g. 1/30 second shutter duration, which is likely to produce significant motion blur, an extremely rapid burst of e.g. three frames is taken at 1/90 second each, and the three frames are overlayed to produce the final image.
In the simplistic case, the frames are simply directly overlayed and the corresponding pixel values are added or perhaps averaged, depending upon the camera. While this helps reduce noise in a sqrt(n) manner for n averaged frames, it does not reduce motion blur.
To reduce motion blur, some cameras apply motion compensation to the burst frames, attempting to move each to the same registration versus the sensor. Typically, this is done using edge detection techniques which identify object edges and other localized regions in the image which exhibit high “activity levels”, meaning lots of difference from surrounding pixels. Most edge detection is done primarily or exclusively with the luminance (Y′) values of the pixels in a YUV or Y′CbCr color space, ignoring the chrominance values. Brightness differences are generally a much better indicator of an edge than are color differences. In these cameras, the motion vector estimation is performed on the entire image (or, more exactly, the entire Nth frame versus the entire 1st frame), or perhaps a portion of the image, to determine a single motion vector, and the entire Nth frame is moved in the opposite vector and composited with the 1st frame.
Most cameras, especially higher priced ones, use a “full frame” shutter which effectively exposes the entire sensor from the same opening time to the same closing time (except of course for the extremely small time differences caused by the motion of the aperture mechanism, which is so small as compared to the exposure duration as to be ignorable for these purposes).
More recently, some inexpensive cameras have been equipped with a “rolling shutter” mechanism which exposes different regions of the sensor at meaningfully different times. For example, a rolling shutter may expose each pixel for 1/90 second but take 1/9 second to expose the entire sensor. In other words, it takes 1/9 second to travel from the first line of the sensor to the last line of the sensor. Because each pixel is exposed for only 1/90 second, each pixel will not have significant motion blur. However, because there is 1/9 second between when the first row of pixels is exposed and when the last row of pixels is exposed, the subject and/or camera may have moved significantly.
What is needed is an improved camera apparatus and method for recompositing multi-frame images, which compensate for the motion-induced de-registration caused by a rolling shutter.